Map Combine Map Task Split HDFS file K 1 , N 1 ( a ) Reduce Task { P 2 } { P 1 } { P 3 }

In parallel query-processing environments, accurate, time-oriented progress indicators could provide much utility to users given that queries take a very long time to complete and both interand intra-query execution times can have high variance. In these systems, query times depend on the query plans and the amount of data being processed, but also on the amount of parallelism available, the types of operators (often user-defined) that perform the processing, and the overall system load. None of the techniques used by existing tools or available in the literature provide a non-trivial progress indicator for parallel queries. In this paper, we introduce Parallax, the first such indicator. Several parallel data processing systems exist. In this paper, we target environments where queries consist of a series of MapReduce jobs. Parallax builds on recently-developed techniques for estimating the progress of single-site SQL queries. It enhances and extends these techniques in non-trivial ways. We implemented our estimator in the Pig system and demonstrate its performance on experiments with the PigMix benchmark and other queries running in a real, small-scale cluster.